Interpreting results of clinical trials: a conceptual framework.

C linical trials are generally designed to test the superiority of an intervention (e.g., treatment, procedure, or device) as compared with a control. Trials that claim superiority of an intervention most often try to reject the null hypothesis, which generally states that the effect of an intervention of interest is no different from the control. In this editorial, we introduce a conceptual framework for readers, reviewers, and those involved in guideline development. This paradigm is based on evaluating a study on its statistical merits (result-based merit) as well as the clinical relevance of the potential treatment effect (process-based merit). We propose a decision matrix that incorporates these ideas in formulating the acceptability of a study for publication and/or inclusion in a guideline. Although noninferiority trials and equivalence trials are other valid trial designs, here we largely focus our discussion on superiority trials. Studies termed “negative” are commonly defined as those where the difference for the primary endpoint has a P value greater than or equal to 0.05 (P ! 0.05) (1), that is, where the null hypothesis is not rejected. These studies are difficult to publish because they are said to be “nonsignificant.” In other words, the data are not strong enough to persuade rejection of the null hypothesis. A high P value is frequently interpreted as proof that the null hypothesis is true; however, such an interpretation is a logical fallacy. A nonsignificant result implies that there was not enough evidence to infer probabilistically that the null hypothesis can be rejected. What is important to keep in mind is that the absence of evidence does not imply evidence of absence (2,3). On the other hand, if a small P value is observed, it implies there is evidence that the null hypothesis is false, which is why much stronger clams can be made when the null hypothesis is rejected. Recall that the null hypothesis (H0) is a stated value of the population parameter that is set up to be refuted. Most often, the value of H0 states that the effect of interest (e.g., mean difference, squared multiple correlation coefficient, regression coefficient) is zero. However, this need not be the case. This point is illustrated in Table 1, a 2 ! 2 statistical inference decision table, wherein what is the true but unknown state of the world is crossed with the statistical decision, thereby generating conceptual definitions for the type 1 and type 2 error rates. In “reject-support” hypothesis testing, by far the most common scenario, H0 is not generally what the researcher actually believes (4), and thus the value of H0 is generally set up to be refuted. When H0 is refuted, that is when P " " (e.g. P " 0.05), strong support exists for rejecting the null hypothesis in favor of the alternative hypothesis, where " is the type I error rate. Until statistical evidence in the form of a hypothesis test indicates otherwise, the null hypothesis is presumed true (1). For example, in a clinical trial with an intervention and a control group, the null hypothesis generally proposes that the intervention and control are equally as effective or ineffective. That is, the population mean of the treatment and control groups is assumed to be the same until an effect estimate (which reaches some prespecified statistical threshold) is observed. What should be clear is that a large P value does not in any way prove that H0 is true. When H0 is not rejected, it implies that either H0 is actually true, reflecting a correct decision (lower left cell of Table 1), or that H0 is actually false but that there was not enough evidence observed to reject H0, a type II error (lower right cell of Table 1). When H0 is rejected, it implies that either H0 is actually false, reflecting a correct decision (upper right cell of Table 1), or that H0 is actually true, but the evidence observed is “unlikely” (which will occur with probability " when the null hypothesis is true; upper left cell of Table 1).